Ensure custody backfill sync couples all responses before importing #8339

eserilev · 2025-10-30T20:01:37Z

Issue Addressed

Custody backfill sync has a bug when we request columns from more than one peer per batch. The fix here ensures we wait for all requests to be completed before performing verification and importing the responses.

I've also added an endpoint lighthouse/custody/backfill that resets a nodes earliest available data column to the current epoch so that custody backfill can be triggered. This endpoint is needed to rescue any nodes that may have missing columns due to the custody backfill sync bug without requiring a full re-sync.

beacon_node/http_api/tests/interactive_tests.rs

eserilev · 2025-10-30T21:19:30Z

#8340 needs to be merged and cherry-picked into release-v8.0. for CI to pass

eserilev · 2025-10-30T22:56:09Z

I'm testing the api changes locally, I'll mark this ready for review once I'm done with that

mergify · 2025-10-30T23:58:06Z

Some required checks have failed. Could you please take a look @eserilev? 🙏

eserilev · 2025-10-30T23:58:32Z

both the custody bakcfill sync bugdix and new api seem to be working just fine on a local sepolia node, tomorrow I'm going to deploy this to one of our internal hoodi nodes and try a custody backfill re-sync using the new api

jimmygchen · 2025-11-02T23:16:50Z

both the custody bakcfill sync bugdix and new api seem to be working just fine on a local sepolia node, tomorrow I'm going to deploy this to one of our internal hoodi nodes and try a custody backfill re-sync using the new api

I'll do this now

book/src/api_lighthouse.md

beacon_node/beacon_chain/src/custody_context.rs

jimmygchen · 2025-11-03T02:33:57Z

When custody backfill is manually triggered, it quickily fills up the beacon.log files with this log:

Nov 03 03:29:07.626 DEBUG Skipping data column import as identical data column exists  block_root: ...

I think we should remove this debug log, as it's a bit excessive when it logs 128 entries per slot on a supernode, and makes it difficult to debug if things go wrong.

mergify · 2025-11-03T05:33:51Z

This pull request has been removed from the queue for the following reason: pull request manually updated.

The pull request #8339 has been manually updated.

You can check the last failing draft PR here: #8349.

If you want to requeue this pull request, you can post a @mergifyio requeue comment.

jimmygchen · 2025-11-03T05:39:30Z

@Mergifyio requeue

mergify · 2025-11-03T05:39:36Z

requeue

☑️ This pull request is already queued

jimmygchen · 2025-11-03T05:44:24Z

@mergify dequeue

mergify · 2025-11-03T05:44:48Z

This pull request has been removed from the queue for the following reason: pull request dequeued.

Pull request #8339 has been dequeued by a dequeue command.

You can check the last failing draft PR here: #8350.

You should look at the reason for the failure and decide if the pull request needs to be fixed or if you want to requeue it.
If you do update this pull request, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio requeue comment.

mergify · 2025-11-03T05:44:48Z

dequeue

✅ The pull request has been removed from the queue `default`

@michaelsproul

…igp#8339) Squashed commit of the following: commit c32c745 Author: Jimmy Chen <[email protected]> Date: Mon Nov 3 16:33:06 2025 +1100 Fix tests. commit d9b2515 Author: Michael Sproul <[email protected]> Date: Mon Nov 3 16:03:46 2025 +1100 Apply suggestion from @michaelsproul commit 7edcb6d Author: Jimmy Chen <[email protected]> Date: Mon Nov 3 15:59:44 2025 +1100 Address review comments commit bb1ace8 Author: Jimmy Chen <[email protected]> Date: Mon Nov 3 15:43:45 2025 +1100 Fix build commit 580b584 Author: Jimmy Chen <[email protected]> Date: Mon Nov 3 15:39:01 2025 +1100 Address review comments. commit 984d1af Merge: 9515494 b57d046 Author: Jimmy Chen <[email protected]> Date: Mon Nov 3 14:47:29 2025 +1100 Merge remote-tracking branch 'origin/release-v8.0' into fork/eserilev/custody-backfill-sync-fix commit 9515494 Merge: 56111a8 55588f7 Author: Eitan Seri-Levi <[email protected]> Date: Sun Nov 2 10:32:01 2025 -0800 Resolve merge conflicts commit 56111a8 Author: Eitan Seri- Levi <[email protected]> Date: Thu Oct 30 14:26:03 2025 -0700 Fix comments commit b213fe5 Author: Eitan Seri- Levi <[email protected]> Date: Thu Oct 30 13:44:11 2025 -0700 Resolve merge conflicts commit 1f72e63 Author: Eitan Seri- Levi <[email protected]> Date: Thu Oct 30 13:33:53 2025 -0700 Fix tests commit a633475 Author: Eitan Seri- Levi <[email protected]> Date: Thu Oct 30 12:56:53 2025 -0700 lint commit e34c881 Author: Eitan Seri- Levi <[email protected]> Date: Thu Oct 30 12:53:56 2025 -0700 resolve merge conflict commit ab04213 Author: Eitan Seri- Levi <[email protected]> Date: Wed Oct 29 16:55:16 2025 -0700 Ensure we've collected all responses before coupling

@michaelsproul

…igp#8339) Squashed commit of the following: commit c32c745 Author: Jimmy Chen <[email protected]> Date: Mon Nov 3 16:33:06 2025 +1100 Fix tests. commit d9b2515 Author: Michael Sproul <[email protected]> Date: Mon Nov 3 16:03:46 2025 +1100 Apply suggestion from @michaelsproul commit 7edcb6d Author: Jimmy Chen <[email protected]> Date: Mon Nov 3 15:59:44 2025 +1100 Address review comments commit bb1ace8 Author: Jimmy Chen <[email protected]> Date: Mon Nov 3 15:43:45 2025 +1100 Fix build commit 580b584 Author: Jimmy Chen <[email protected]> Date: Mon Nov 3 15:39:01 2025 +1100 Address review comments. commit 984d1af Merge: 9515494 b57d046 Author: Jimmy Chen <[email protected]> Date: Mon Nov 3 14:47:29 2025 +1100 Merge remote-tracking branch 'origin/release-v8.0' into fork/eserilev/custody-backfill-sync-fix commit 9515494 Merge: 56111a8 55588f7 Author: Eitan Seri-Levi <[email protected]> Date: Sun Nov 2 10:32:01 2025 -0800 Resolve merge conflicts commit 56111a8 Author: Eitan Seri- Levi <[email protected]> Date: Thu Oct 30 14:26:03 2025 -0700 Fix comments commit b213fe5 Author: Eitan Seri- Levi <[email protected]> Date: Thu Oct 30 13:44:11 2025 -0700 Resolve merge conflicts commit 1f72e63 Author: Eitan Seri- Levi <[email protected]> Date: Thu Oct 30 13:33:53 2025 -0700 Fix tests commit a633475 Author: Eitan Seri- Levi <[email protected]> Date: Thu Oct 30 12:56:53 2025 -0700 lint commit e34c881 Author: Eitan Seri- Levi <[email protected]> Date: Thu Oct 30 12:53:56 2025 -0700 resolve merge conflict commit ab04213 Author: Eitan Seri- Levi <[email protected]> Date: Wed Oct 29 16:55:16 2025 -0700 Ensure we've collected all responses before coupling

jimmygchen · 2025-11-03T07:04:45Z

@Mergifyio requeue

mergify · 2025-11-03T07:04:50Z

requeue

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

N/A Includes the following unmerged PRs: - #8344 - #8335 - #8339 This PR should be merged after all above PRs are merged. Co-Authored-By: Jimmy Chen <[email protected]> Co-Authored-By: Jimmy Chen <[email protected]>

eserilev requested a review from jxs as a code owner October 30, 2025 20:01

eserilev added bug Something isn't working syncing HTTP-API das Data Availability Sampling fulu Required for the upcoming Fulu hard fork v8.0.0 Q4 2025 Fusaka Mainnet Release labels Oct 30, 2025

eserilev changed the title ~~Custody backfill sync fix~~ Ensure custody backfill sync couples all responses before importing Oct 30, 2025

eserilev commented Oct 30, 2025

View reviewed changes

beacon_node/http_api/tests/interactive_tests.rs Show resolved Hide resolved

eserilev changed the base branch from unstable to release-v8.0 October 30, 2025 20:38

eserilev force-pushed the custody-backfill-sync-fix branch from ffc5474 to 53a79cb Compare October 30, 2025 20:40

eserilev added 5 commits October 30, 2025 13:42

Ensure we've collected all responses before coupling

ab04213

resolve merge conflict

e34c881

lint

a633475

Fix tests

1f72e63

Resolve merge conflicts

b213fe5

eserilev force-pushed the custody-backfill-sync-fix branch from 53a79cb to b213fe5 Compare October 30, 2025 20:44

Fix comments

56111a8

eserilev added the ready-for-review The code is ready for review label Oct 30, 2025

mergify bot added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Oct 30, 2025

Resolve merge conflicts

9515494

jimmygchen reviewed Nov 2, 2025

View reviewed changes

book/src/api_lighthouse.md Outdated Show resolved Hide resolved

jimmygchen reviewed Nov 3, 2025

View reviewed changes

beacon_node/beacon_chain/src/custody_context.rs Show resolved Hide resolved

mergify bot added the queued label Nov 3, 2025

mergify bot added a commit that referenced this pull request Nov 3, 2025

Merge of #8339

5e7e237

mergify bot mentioned this pull request Nov 3, 2025

merge queue: embarking release-v8.0 (b57d046), #8344 and #8339 together #8349

Closed

6 tasks

Fix tests.

c32c745

mergify bot added queued and removed queued labels Nov 3, 2025

mergify bot added a commit that referenced this pull request Nov 3, 2025

Merge of #8339

01c9d7e

mergify bot mentioned this pull request Nov 3, 2025

merge queue: embarking release-v8.0 (b57d046), #8344 and #8339 together #8350

Closed

6 tasks

mergify bot added a commit that referenced this pull request Nov 3, 2025

Merge of #8339

66be781

mergify bot mentioned this pull request Nov 3, 2025

merge queue: embarking release-v8.0 (b57d046), #8344, #8339 and #8335 together #8351

Closed

6 tasks

mergify bot added dequeued and removed queued labels Nov 3, 2025

jimmygchen mentioned this pull request Nov 3, 2025

Release v8.0.0 #8352

Merged

mergify bot added queued and removed dequeued labels Nov 3, 2025

mergify bot added a commit that referenced this pull request Nov 3, 2025

Merge of #8339

dea4508

mergify bot mentioned this pull request Nov 3, 2025

merge queue: embarking release-v8.0 (25832e5) and [#8335 + #8339] together #8354

Closed

10 tasks

mergify bot merged commit 5d0f8a0 into sigp:release-v8.0 Nov 3, 2025
35 of 39 checks passed

mergify bot removed the queued label Nov 3, 2025

Ensure custody backfill sync couples all responses before importing #8339

Ensure custody backfill sync couples all responses before importing #8339

Conversation

eserilev commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue Addressed

Uh oh!

Uh oh!

eserilev commented Oct 30, 2025

Uh oh!

eserilev commented Oct 30, 2025

Uh oh!

mergify bot commented Oct 30, 2025

Uh oh!

eserilev commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jimmygchen commented Nov 2, 2025

Uh oh!

Uh oh!

Uh oh!

jimmygchen commented Nov 3, 2025

Uh oh!

mergify bot commented Nov 3, 2025

Uh oh!

jimmygchen commented Nov 3, 2025

Uh oh!

mergify bot commented Nov 3, 2025

☑️ This pull request is already queued

Uh oh!

jimmygchen commented Nov 3, 2025

Uh oh!

mergify bot commented Nov 3, 2025

Uh oh!

mergify bot commented Nov 3, 2025

✅ The pull request has been removed from the queue default

Uh oh!

jimmygchen commented Nov 3, 2025

Uh oh!

mergify bot commented Nov 3, 2025

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

eserilev commented Oct 30, 2025 •

edited

Loading

eserilev commented Oct 30, 2025 •

edited

Loading

✅ The pull request has been removed from the queue `default`